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The Council of Chief State School Officers (CCSSO) is a nationwide non-proHt 
organization of the 56 public officials who head depanments of public education in every 
state, U.S. Territory, and the District of Columbia. CCSSO seeks its members' consensus 
on major education issues and expresses their views to civic and professional organizations, 
to federal agencies, and to Cdngress, responds to a broad range of concerns about 
education, and provides leadership on major education issues. 

Because the Council represents the chief education administrator in each state and 
territory, it has access to the educational and governmental establishments in each state 
and the national influence that accompanies tnis unique position. CCSSO forms coalitions 
with many other educational organizations and is able to provide leadership for a variety of 
policy concerns that affect elementary and secondary education. Thus, CC^SO members 
are able to act cooperatively on matters vital to the education of America's young people. 

The State Education Assessment Center was founded by CCSSO in 1985 to provide a locus 
for leadership bv the states to improve the monitoring and assessment of education. This is 
a report of the Assessment Center's National Assessment Planning Project. 
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NATIONAL ASSESSMENT PLANNING PROJECT 

The National Assessment Planning Project, under the auspices of the Council of 
Chief State School Officers (CCSSO), was CTeated by a consortium of eighteen national 
organizations interested in education and in exploring the feasibility of expanding the 
National Assessment of Educational Progress (NAEr) in order to produce state-by-state 
comparisons of student achievement 

The project is governed bj^ a Steerins Committee. Each member was appointed on 
the recommendation of an organization in me consortium. This publication was conveyed 
to the Department of Eduction (ED) and to the National Assessment of Educational 
Progress on the review and approval of the Steering Committee. The publication, however, 
does not necessarily reflect the views of each of the associations in the consortium. 

The project was supported by Grant No. SPA- 1549 from the National Science 
Foundation (NSF) with fundspartly from NSF and pardy from the National Center for 
Education Statistics m the U.S. Department of Education tiirough an inter-sovemmental 
transfer from NCES to NSF. This publication, however, does not necessariw reflect the 
views of either agency. The interest of the two ajKencies in this project and tneir willingness 
to provide joint support is greatly appreciated. Tne sunport of NGbS and NSF made 
possible this unique and vital step in the process of makmg NAEP more useful for 
policymaking at the state and load level. 
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I. INTRODUCTION 
PURPQSE/HISTORirjlL PERSPEmVE 

Interest in state level comparisons is rising in many auariers in the belief that better 
state-comparative information about student achievement man is currently available would 
facilitate the improvement of education in the individual states. In 1984 a majority of 
CCSSO members sup^rted the development of a system of student assessment that would 
provide state compansonsS and in 1983, the members endorsed the expansion of NAEP as 
the most feasible way of providing these comparisons. During the 1986 national 
assessment, two individual states, Wyoming and Georgia, contracted with NAEP to conduct 
in-state, concunent assessments and to provide state-to-nation comparisons. Also in 1986 
and in 1987, groups of southern states, in a project coordinated by tne Southern Regional 
Education Board, contracted with NAJEP to conduct state-level assessments^. They were 
provided reports comparing achievement amona those states. Some of the rising interest 
can be attributed to governors. A 1987 report nom the National Governors' Association 
entitled Results in Education presented a number of comparative education indicators and 
displayed a blank column for achievement, clea^ expressing the intent to fill that column 
in tiiture years with achievement data . In a 198/ report, a national group appointed by 
Secretary of Education William Bennett and chairea by former Governor ortennessee 
Lamar Alexander made a series of recommendations on the future of NAEP^. A major 
recommendation was that NAEP should be expanded to provide state-l^-state 
comparisons. 

This rising interest is not without its critics. Some are worried that Federal, state, 
and local policymakers mcf misuse the data, making inappropriate inferences and drawing 
unwananted cause-and-effect conclusions. Fears are expressed that the test will be very 
influential, and, with that influence, foster a national curriculum. StiU others fear that the 
compromises that might be made on objectives will result in an assessment that measures 
the least conunon denominator and discourages badly needed curriculum reform. 

Designing a national assessment that would not only be constructive but also 
minimize potential disadvantages is the purpose of this National Assessment Planning 
Project The project has made recomissndations that answer two questions of major 
interest to state and local educators and policymakers, who have been asked for the first 
time whether th^ want a report card for 'ibeir state. 

They will likely want to know what mathematics objectives (knowledge, skills) the 
assessment will measure and whether the objectives are more or less compatible witn what 
they believe the schools in their states are tiyins to teach or believe shoi'ld be taught. The 
topic of a sepuAte publication is the mathematics objectives for the 199U math 
assessment^ That repon describes the basis for the development of test item specifications 
and for iteiM on the 1990 math assessment 

They will also want to know how the achievement data on students in a state will be 
reported. A secoud report irakes recommendatioiis on how state achievement data from 
the 1990 math assessment should be measured and reported by NAEP, bow comparisons 
should be reported, and on several policy issues relatM to conducting state-level 
assessments^. 

ORGANIZATION OF THE RRPOirr 

This is the third report of the National Assessment Plannins Project. It is the report 
of the consensus planning process used in the Project During 1987-88, the National 
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Assessment Planning Proiert was conducted by a consortium of eighteen organizations 
brought together by the Council of Chief State School Officers. Tb6 project used 
consensus planning procedures to arrive at its recommendations. Th& report is on the 
consensus procedures used, and it makes recommendations to the U.S. Department of 
Education National Center for Education Statistics and to the NAEP grantee on 
conducting consensus planning in the ftimre for the National Assessment 

This report is organized in four sections. The first section is background 
information on the National Assessment of Educational Progress and on the planning 
project The second section is a discussion of how the demands placed upon the planning 
of the assessment shift as it changes to a state-by-state program from a national pronam. 
The third section describes the consenrjs planmng procedures used in the project. Finally, 
the fourth section presents recommendations, observations, and lessons learned from the 
project 



II. NAEP BACKGROUND 

INTRODUCTION 

Since the late 1960*$, the National Assessment of Educational Progress (NAEP) has 
administered tests to a representative national sample of 9, 13, and 17 year old students 
and issued reports of what children know and gati rfn in reading, mathematics, writing, 
saence, and other subjects. The assessments have been based on a sample of 25,000 
smdents at each age level, enough to provide data that are reliable on a national and broad 
regional basis. The sample, however, has not been large enoush to provide measures of 
smdent performance at the state level, measures that would allow state-by-state 
comparisons. 

There was little interest in comparing student performance in one state to another. 
That has chansed dnmaUcdJly in the last few years, as states have been asked to assume 
more responsioility for education and as the need for information to monitor education has 
increased. 

As described above, the states, represented by the Council of Chief State School 
Officers (CCSSO), adopted a poliof that would provide for state-to-state achievement 
comparisons, and sevenl states inmvidualW and collectively have already moved toward 
using NAEP for assessing and oonmaring toeir schools' performance. In 1987 the U.S. 
Department of Educati(m awarded a grant for the operadon of NAEP until 1990 to the 
Educational Testing Savioe which isduded die possilMiity diat some part of die 1990 
assessment teit a luge enough sample of students in each state to provide state level data- 
or at least is those states that jsaot to be assessed. This spring. Congress re-authorized 
NAEP witk new piovisions for state-by-state assessment on a pilot basis in die 1990 and 
1992 NAEP cydw. *^ 

Whfit also has changed is die need for NAEP to be sensitive to the curriculum 
objectives of die ini&ridualstates and school districts as it becomes a state-level 
assessment In die past, diat has not been cnidal; die assessment could reflect a general, 
national consensus. It however, education leaders in a particular state are faced widi the 
decision of whether to participate in a state-con^iarative assessment th^ need assurance 
that the knowledge and skills diat the assessment measures are oompatiole with the 
knowledge and slolls diey are attempting to teach to their smdents, or at least, die 
knowledge and skills diat the leaders beueve should be taught They will also want to know 
how the dau will be analyzed and reported. A Single number only showing diat smdents in 



a state with a high proportion of low-income families do not perform as weU as states with 
more affluent students is hardly enlightening. Other variables that are included in the 
analysis, and the formats for reporting data, will be very importac*. if the state-to-statc 
comparl'ons are to be constructive and fair. 

It is around these issues-the knowledec and skills that should be assessed and the 
method of analyzing and reporting state-level data-that the consensus planning process 
used m the National Assessment Planning Project was designed. Recommendations on the 
content of the assessment and on the methods for analyzing and reporting results were to 
be produced through a "consensus process" that was sensitive to the interests and concerns 
ot tue states and local districts and of various other constituents. What can be said about 
the consensus procedures used m that process-their strengths and weaknesses and the 
lessons that can be learned from the experience for procedures used in the future? 

FEDERAL LE GISLATION 

The legislation previously authorizing NAEP as a national testing program required 
a consensusplammig process for its design: "Each learning area assessment shall have goal 
statements devised through a national consensus approach, providing for active 
partiapation of teachers, curriculum specialists, subject matter spcaalists, local school 
J^™^"ftrators, parents, and members of the general public ..." (Public Law 98-511, Section 

403(e)) 

A virtually identical provision is included in the new law re-authorizing NAEP and 
expandmg it to a state-by-state program: "Each learning area assessment shaS have goal 
statements devised through a national consensus approach providiiu for active 
participation of teachers, curriculum specialists, local school administrators, parents and 
concerned members of the general public ..." (PuHic Law 100-297, Section 3403) 

(Copies of these laws are included in Appendices C and D) 

. . The participatory process that was afBrmed as the principle for arriving at consensus 
m the past becomes all the more crucial as uses of NAEP for comparisons and 
poUcymajang are expanded, and as various state and local constituencies develop a greatc 
stake m the results. ^ 

CONSENSUS PLANNING PROCESS USED BY NAEP 

Within die constraints of available resources, the NAEP grantee has attempted to 
use a planning process desimed to yield a national consensus on the goals and objectives 
for each assessment which is fsithfiil to the intent of the legislation. For each subject-area 
asseument» one or mofe grantee's staff have been designated as responsible for carrying 
out this coiMinis piaim^ process. With the approvalof the project's Assessment Pohcy 
Committee (AFC), staff have formed Learning Area Committees to develop the objectives 
for the assesnuent Consisting usually of 7-9 members, these committees have been 
constituted to indude: 

0 university-based soedalists in the teaching or learning of the subject; 

0 national, state, or local curriculum specialists in the subject; 

0 prominent teachen of die subject; 

0 local school administratois, generally selected because of some professional 

emerience and interest in the subject; 

0 scbolan in the subject being assessed; and 

0 policymakers or representatives of the lay public 
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• committees generally have worked from the objectives of prior asM«meiit« 
rcfimn^ and updatm^ them, revising them to reflect more contemwjrarv I^^^ 
the subject, or changig them to reflect current interests in Ste^ SSSJf T^l 
committees produced a set of ob ectives which were then ciraSated broadWfS? wr^ent 
Reviewers were asked, among other things, to comment on wKr Se SSivS 
departed substanti^y from prevalentlSctice. TTiesc commentswere c^SfdSin 

A^^t^Sf* the objectives, which were EibStted to Si^APC 
^JT^' approved, the objectives served as a point of reference for 

a^s?^^.JS development committees is they wrote questiCi^d iteSL f^4e 

«k;.^ J5® assessment was a transitional one for NAEP. Until then, develooinff 
objectives was a one-stage process conducted at the national level as dScSaS Oni. 

school system wS Scd to1?Se obSSVes 

Commi«l'5?n1^'?' ^ ^^^^^^ N/^P gnSe^aSd^d a Stete AdS 

Committee m addition to the overall Learning Area Commttee This wlfTtnnrf^^u^.^ 

comprehensive and deeper consideratiSSby?tat« 

assessments conduc;^ at the state level, it Is presumed that^Se gwenSIici Md assessment 

reading'^S^^^^^ 1988 

?nT°r Pjf^ ^ and list the indiviSab^Kei^hJSSM^^^ 

kmds of partiapants who have been represented. uwoivea. snowing tne 

III. HOW STATC-BY-STATE ASSESSMENT CHANGES TOE DEMANDS PLACED ON 

TOE PLANNING PROCESS 

There are several conditions and features that define the consensus aooroach 
wtrS^ ?9r°°** "^^"^ P'°«^ ^ the'lSISK 

° K^'SSiiShiSS^f educational unit (school, district, state) that has 

° IIl***lP^^^.«»?»««n»entdocsnotnccdtobcdosclyaUg^^^ . 
educational objectives of aa "accountable unit" y ougacu wiui me 

0 The law requires participation be voluntary. 

0 Ayeement to participate in a national assessment is typically based on the 
e^prsement of the state :;upermtendent and on the agreement and 
wimngness of local district superintendents and principals of schools selected 
m the sampling process. TTiis is based on the belief thatNAEP is valuable or 
on the desire to be cooperative, not on the extent to which th e test matches 
tne local curriculum. 



o Schools are sampled rarely, only a few students are involved in each school, 
NAEP provides someone to give the tests, and time out of class is brief, so 
participants have little at stake in the procedures used. 

STATE ^ASSESSMENTS AND COMPARISONS CHANGE THE CIRCUMSTANCES 

With the expansion of NAEP to produce state comparisons, the needs which the 
consensus process must meet are different, and material cnanses m ist be made in the 
process^ compared with when the |>rogram was simply nations! in scope. The change in the 
demands is so fundamental that it prompted the blue*ribbon panel chaired by former- 
Governor of Tennessee Lamar Alexander on the expansion of NAEP to come up with a 
whole new model of governance for the project 

In state-comparative assessment, the changed circumstances meriting such a 
substantial restructuring of the governance and consensus planning process are as follows: 

0 In state assessment, there ar^: identifiable units which are accountable and 

geople V ho feel re^nsible for the performance of these units: State 
uperintendents, State Boards of Education, Legislators, Governors, and, 
inoirecUy, the policymakers and administrators of local districts which make 
up the states' school systems. 

o Locals feel that results of state assessments may become the basis for a state- 
level change in what expected of them. Though local accountabilitv is 
indirect, it is nevertheless strong. As Lillian Bama, then Superintendent of 
the Albuquercjue Schools and a member of the Planning Project Steering 
Committee said, *lf someone is concerned about how I^w Mexico is domg, 
they are going to turn first to Albuquerque to try to bring up the state's 
average. 

o With a **state report card," state and local officials must be concerned about 
the aiimment or the test to their curriculum. It is fair and reasonable to test 
and hold states and loot! districts accountable qqIx for a body of knowledge 
and skills that they agree is important 

o What the test measures becomes a significant factor in whether states and 
locals will be willins to partidptte in this voluntary activity. Since it is 
voluntaiy, the decision to enter will be based in part on the nature of the 
game, as defined what will be tested. 

0 With state report cards and a larger sanq>le, more schools and students are 
affected^ state and local staff become more involved in administration of the 
program, and the conditions under which results are obtained and used for 
comparisons become more critical; thus, the procedures of the assessment 
become more important 

The new governance structure recommended by the Alexander group expanded the 
planning and objective-^tting of the project; distinguished it from the exercise- 
development, data-collection, and analyus activities; and placed governance and policy- 
setting under an independent National Assessment Governing Board with policy-setting 
authority. The reasons for this were to create a stronser, more independent governance 
body that would be given over to constituencies including the states, which had ^J^^^t 
stake in the conduct and ramifications of the expanded, state-by-state program The 
primary responsibility of this Board, as conceived by the Alexander panel, was to provide 
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for the deepened and broadened consensus-development process required for state- 
comparative assessment, and the Board was envisioned and intended as a means of 
participatoiy self-governance of the project, ijy the field, through such a consensus- 
development process. The National Assessment Planning Project was intended to 
approxunate this kind of self-governance and planning. 

In addition to providins for a deeper consrisus by states and others concerning what 
is tested, the changes Drought oy state-by-state assessment bring other, new governance 
needs. The greater salience of the promm will brin{| not only more attention to 
educational programs in the states andlocaiities; it will also brins more attention to the 
procedures and design of NAEP itself; and the governance boara must protect and insulate 
the integrity of the program, while still being open to appropriate insights and 
improvements. 

Finally, the expanded program will require conmiitment from local school districts 
to participate, and the planning of the project must support and encourage such 
commitment Since there is no immediate incentive for locals to participate, and since 
there are several disincentives, the jgovemance process must cultivate sevend, indirect 
bases for local participatl;>a: the ideal of the quality of the data; involvement in planning 
the program (and hence, development of ownership); and recehong data back after 
participation, even if its utility may be limited to servmg as a check on standardized testing 
prograir^ 



IV. CCSSO CONSORTIUM NATIONAL ASSESSMENT PLANNING PROCESS 

1987-88 

EUBEQSE 

The targets or topics of the planning done in the National Assessment Planning 
Project were twofold: to develop subject-area objectives in mathematics for the first state- 
by-state assessment in 1990 and to recommend the procedures to follow and the analytical 
models to be used in reporting results of the 19!90 trial state assessment. 

The goal of the consensus planning process was both educational and political. At a 
practical or strategic level, it was desirable to plan through a process that enaoled those 
wiUi a stake to air their views, develop ownership of the plans, and get on board with the 
emerging state-level NABP program. 

EdiiCitional(|r, a vision had emerged for the planning process in which it would be 
conducted io at to yield a more desirable product-plans that truly represented the best 
thinking of die fielo as to what was inqMrtaut to measure in the subject and how state-by- 
state assessment should be done. This vision had emerged in 1986, during the deliberauons 
of Uie Alexander Study Group, when that group considered how a NAEP expanded to 
produce state-level renilts should be governed. Ihen, it was felt that the project should be. 
governed by those with title greatest stake in the outcomes, and that governance should go 
beyond pohcy advice to include planning, setting the substantive objectives, designing the 
k^ features of the program and planning its procedures. It was felt that planning should 
be based on deep and extensive participation by teachers, policymakers, subject specialists, 
technicians and aityone who could bring visionary thinking to bear on the design of the 
assessment It was envisioned that this process would be jvofessionalfy exdtijis and that 
the assessment plans would reflect not j[ust prevalent practice in the field, but the best that 
could be said in our nation about teachmg, learning, and assessing a subject at any point in 
time. 



For the National Assessment Planning Project, development of the mathematics 
objectives for the 1990 state-by-state assessment v/as conceived as requiring a consensus of 
the field as to the subject-matter to be tested, including some fom of consensus or 
reconciliation of the subject ' tter reconmiended to be assessed with current state and 
local curriculum policies in l. jematics. 

Reconunendations on assessment procedures and the models to be used to analyze 
and report results were viewed as requiring a different sort of consensus. A number of 
technical issues were understood to be involved in the expanded assessment that would 
require resolution through consultation with the best technical experts in the field. States 
were seen as possessing technical experience and expertise which should be solicited, 
distilled, and used to guide the design of the state-b;-state assessment, but they were also 
seen as having a stake in mai^ aspects of how the data will be analyzed and reported: a 
decision made one way or another (such as how to use demoaraphic data to interpret 
results) could affect how states come out in the assessment So, state staff as well as other 
experts were asked to advise on technical matters, and states were given an opportunity to 
comment on those design decisions which had great political and educational miplications 
'or them. 

WORK PLAN AND SCHEDULE 

The orij^nal work plan and time schedule for the planning process were developed 
by staff and reviewed, modified and approved by the project Steermg Conunittec at its 
August, 1987 meetins. lliis provided for recommenoations be completed in February, 
1988. The schedule for the National Assessment Plannir Project was determined largely 
1^ the need to co olete planning in time to inform the dw xlopment of exercises for the 
1990 assessment, oich had to begin in April, 1988. This was an extraordinary constraint 
which made an adequate planning process impossible in some ways. Diagrams showing 
these schedules are mcluded in the appendix m order to explain the processes used in this 
project; these schedules are not recommended as models for the future, because they were 
too short for adequate completion of most tasks. 

At its initial meeting in Ausust, the project's Steerins Conunitttc nad adopted a 
policy statement on die purpose of state comparisons and the conditions that should be 
met, as follows: 

The purpose of state level student achievement comparison is to provide data on 
student performance to assist policymakers and educators to vrark toward the 
ini^rovement of education. Such data can be useful by encouraging and 
contributing to .a discussion of the quality of education and the conditions that 
determine It 

State-comparative achievement data are useful if they: 

0 Represent performance on a consensus of what is important to learn; 

0 Use sound testing and psychometric practices; 

o Use procedures that minimize intrusion into instructional time; 

o Take into account different circumstances need', that the states lace; and 
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0 Are associated with features of the school systems that can be improved by 
policymakers and educators. 

The principles and parameters guided the two consensus planning processes. 

The process used to attain each planning purpose is described below. 
Mathematics Ohiectlvea 

The activities which resulted in mathematics objectives for the state-by-state 
assessment were patterned in part after the consensus process that evolved over the years 
in planning prior national assessments in response to toe language in Public Law 98-511, 
Section 4CB (e) authorizing NAEP. In addition, however, the o^ectives were developed in 
a manner that was based on the recomition that the 1990 assessment in mathematics will 
provide state-by-state comparisons of >tudent achievement Because state report cards, as 
well as "The Nation's Report Card," would result from this assessment, the process was 
expanded considerabfy beyond recent practice to ensure that careful attention was given to 
the formal mathematics objectives of states and a sampling of local districts, and to the 
opinions of practitioners at the state and local levels of what should be assessed. The 
process, carried out between August, 1987 and March, 1988, had the following features: 

o A Consortium Steering Committee with members recommended by each of the 18 
national organizations representins poli^makers, practitioners ana citizens met, 
modified, and ^>proved the overall plan of work. 

o A Mathematics Objectives Committee was created to draft a set of recommended 
objectives. Its membership consisted of a teacher, a local school administrator, sfite 
mathematics education specialists, mathematicians, parents, and citizens. It met 
once for preliminaiy planning to consider and determine what information it would 
need prior to a major work session in December, 1987. For its review and 
consideration, the committee was provided the following: 

o Math Objectives 1885-86 Assessment (NAEPV 

o Curriculum and Evaluation Standards for School Mathematics (National 
Council of Teachers of Mathematics). 

o Content analyses of state and local mathematics guides produced under 
subcontract 1^ the Education Products Information Exchange. 

o Suggestions solicited from state mathematics specialists. 

0 A report on "Issues in the Field" based on telephone interviews with leading 
mathematics educators. 

o A draft framework provid<*d by a mathematics objectives subcommittee. 

0 At the December meeting, the Mathematics Objectives Committee took up the sub- 
committee's draft, enhancing and modifying it 

0 In subsequent weeks, the draft report was edited for form, sample questions were 
added, and a com was mailed to each state department of education mathematics 
specialist in the utty states. 
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o These individual specialists in each state were requested to convene a committee of 
state and local people to respond to the objectives. Tliose committees reviewed the 
draft report and returned conmients and suggestions to the project st^. 

o A copy of the draft report was sent to, and comments received from, twenty-five 
national mathematics educators and mathematics scholan. 

o The reactions were given to the Mathematics Objectives Conmiittee, which met a 
third time in late February. 

o Modifications were made in response to the comments, and the final draft report 
was reviewed, modified, and approved by the Project Steering Committee in mid- 
March. 

PROCEDURES. ANALYSIS. AND REPORTING 

A second target of the planning project was to consider how the achievement data 
should be collected, how data on students m a state should be presented, and how state-to- 
state comparative data should be reported. However popular with some, a simple number 
showing only that the average achievement of students in one state is hi^er or lower than 
that of students in another state is not enough. Recommendations were made on how state 
achievement data from the 1990 math assessment should be measured and renorted by 
NAEP, how comparisons should be reported, and how several poliq^ issues related to 
conducting state*level assessments should be resolved. 

The pi ocess which produced these recommendations was desimed principaUy to 
result in a broad, field*based consensus on the issues of e3q)anding NAEP to $tate-level 
assessments. In part, the . ^roach is reflected in the consensus process developed over the 
years NAEP to select the objectives or knowledse and skills tnat an assessment will 
measure. That consensus process is responsive to tne Federal law creating the National 
Assessment, but since the reconunendations address analysis and reporting matters and 
several assessment poli^ issues, rather than the content that is tested, a national consensus 
process was not tecnnicaUy required Public Law 98-511 vested authority for setting 

B)lides and approving reports with the NAEP Ass»»ment Poli^ Committee, 
evertheless, an early determination was made to engage many policymakers, technicians, 
and scholars from across the country in an extensive process ot consultation and 
collaboration on these procedural and desi^ matters. This determination was based on 
the belief that such a process was likely to identify significant problems with designing 
'^state report cards»" as well as to provide constructive ideas. 

The report was.drafted under the project*s Procedures, Analysis and Reports (PAR) 
Conmiittee» but many othen were involved, particularly the directors of assessment 
programs in eadi of the states. Many of these directors made recommendations during the 
plaiming procesi; they also responded to drafts with helpful suggestions. The process, 
carried out between August 1987 and March 1988, had the following features: 

o The project Steering Committee modified and q>proved the overall plan of work. 

o The Procedures, AnaWsis and Reports Committee was created to draft a report. Its 
membership consistecf of policymakers, state and local district assessment 
specialists, and scholars in the field of assessment 

o The Committee met once for preliminary planning to consider its charge and 
determine what iitformation it wished to have prior to its major work-session in 




December. For that major work-session, the Committee was given many 
documents, including the following: 

o G}Dies of conespondence from state directors of assessment raising issues 
and suggesting ways that comparative data should be reported. 

o An options paper entitled "Alternate Ways of Reporting State-by-State 
G>mparisons" prepared for the Committee under a contract 

o Several recent publications on education indicators from CCSSO, the RAND 
Corporation, and others; lists of conmion student background questions from 
a variety of studies, including the National Education Longitudmal Study of 
1988, Schools and Staffins Survey, Conunon Core of Data, and High School 
and Beyond; and the pubGcadon, The Underachieving Curriculum 
(International Study of Math Achievement). 

0 A background memorandum from the Southern Regional Education Board 
reviewing the procedures followed in eight southern states, in which school 
district employees administered NAEP tests in a state-by-state program. 

o A report provided by the National Association of State Directors of Special 
Education on the inclusion of special students in NAK*. 

o During its December meeting, the Conunittee made tentative decisions about the 
recommendations it wished to make. 

o In subsequent weeks, a report was drafted by project staff based on those decisions, 
and copies were sent to state education a^enaes and selected state polii^makers, 
local district educators, and scholars. Projea staff discussed this report with the 
assessment subcummittee of the Committee on Evaluation and Information Systems 
(CEIS) of the Council of Chief State School Officers in a special meeting in January. 

o All comments received on the draft were provided to the Procedures, Analysis, and 
Reports Committee, which met again in late February. The Committee reviewed 
the comments and completed its report 

o The Committee's report was reviewed, modified and adopted by the Project 
Steeripg Committee. 



V. RECOMMENDATIONS AND OBSERVATIONS ON THE PROCESS 
GENERAL ISSUES IN THE CONSENSUS PLANNING PROCESS 

1) The proccM used hy the National Assessment Grwaminy Board to plan NAEP 
should have Ae following features: 

0 It should be partidpatofy. providing for democratic, bottom-up involvement 
of the field and of the constituencies affected by NAEP in determining its 
features. 

0 It should be visionary, seeking out and tapping the appropriate and best 
thinking in the nation (and the world) on the unportant aedsions faced in 
planning the assessment: what subject-matter to test at each grade level, 
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what analyses to perform in interpreting results, how best to capitalize on 
sampling and scaling methodolo^, and how logistically to cany out a sound 
and fair assessment. 

0 It should be iterative or interactive to allow suggestions and advice to 
influence one another and to CTeate a planning process which evolves 
positively. 

o It should be structured and guided in advance by the policies and principles 
that are needed to conduct me planning process: what authority do 
committees have? Under what constramts are they operating? In what 
direction should they head? 

0 It should be explicit in its rules, principles and procedures; these features 
should be set openly and explicitly, and they should be communicated 
effective^ to the field to enhance the participatory qualiqr discussed above. 

0 It should be stable, following a viable cycle to its completion, adhering to 
predictable procedures as assessment <ydes repeat themselves, and 
amending tnese procedures only when there are important reasons to 
change them. 

0 It should be long enough to allow the values articulated above to be attained: 
participation by the field; thoug^tfiil consultation for the best resolution of 
important issues; and plamuig in a way that is explicit, iterative, and 
strucmred by the necessary policies. 

0 It should be su pported by adequate devotion of time by the National 

Assessment Governing Board and its constituent committees, by adequate 
and appropriate-qualined staff who support the process in a professional but 
neutral manner, and by adequate operational resources to support the 
expenses entailed in pianniiu by the field a yovemancg process which does 
not have these three fands of support gnminittitd to it at the outset bv 
Congress and bv the U.S. Department of Edugatinn should not proceed. 

The discussiona of consenn is committees chnnlH managed in a valucfree wav. so 
as to he frecfl owing. to encourage opinions, spontaneity and creatmtv. and to avoid 
nirtailiny or intimidating the partidpants. 

At some points, the thinldng of eveiyone involved in a conunittee planning process 
nuqr fiher or go astray. When this happens, either of two courses can be taken: one 
is to iet the group work throush the process, assuming that the process of self 
gofemnce and consensus-building will work itself out The other course is to 
nr ^f in f ge the plamiing process somehow. This can be done, and it should be done 
throinh the gentle guidance of a competent but unbiased professional staff, a staff 
knowledgeable enough to know that toe process has gone astray, but neutral and 
able to move the group along, though not toward any particular end. 

The consensui t planning process is an activity that should be mumallv educational 
for those involved. 

At times, the thinking of some partidpants will be more sound or creative than that 
of others; the latter will learn from the former, and the roles may be reversed at 
another time in the process. 
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4) Values and constraints governing the process should be stated up front 

In particular, judgements or values should be set explicitly at the outset: What will 
constitute "consensus"? How much innovation will be tolerated? How much detail 
is to be sought in the recommendations? How will differences be handled-will they 
be resolved, ignored, or expressed as a minority view? What "givens" from outside 
constrain the process? 

5) Chances in the basic structure or rules of the mns ^nsus process while it is going nn 
must be avoided. 

Changes while the process is goins on can disenfranchise the process itselt obviating 
ordevalui ^it To some extent, the governance board can avoid this by making 
sure that the structures and ^ound-rules it controls are protected, left m place 
through the process until a tune at which they can be changed. Others involved- 
Congress, the Seaetaiy of Education, the Assistant Secretary for Research and 
Improvement, the Commissioner of NCES, and the staff of NCES-must also protect 
the integrity of each consensus planning process, by allowing it to proceed through 
one set of ground-rules before changing the rules. These individuals actually must 
help insulate the process from disruptive changes that could undermine it 

6) Solicitation of comment hv formal cntnmiitees riyr^nring tho. fi^!H ^j^hfif nn 
content or procedures, is needed nnh, in response to draft reconini«.nrfatinns 

In November, suggestions on the subject-matter framework were requested from 
state department math specialists ratner than a state committee, and a similar 
solicitation was made of state assessment program staff for comments on procedural 
issues. This wpeared adequate, as opfostd to setting up and soliciting comments 
from a formal committee in each state at that point as well as later, when the 
recommendations were oon^lete. Formal committees should be set up for review 
of reconmiendations in each subject to be assessed in states and in other 
constituency groups such as speaaUst teacher orsanizations. At earlier stages, 
suggestions can be sought from staff representatives of these constituencies. 

7) With whom to work in stata departments of education and in other constituencies 
must he decided r«ri»fiiHy 

Chief state sdiool ofBoers need to be kept informed throughout as do the officers or 
othv rq>reseittativei of other ma^r stakeholders. Communicating directly with the 
subftct q)ecialisti in SEAs for initial suggestions and to set up state committees for 
re^ome to draft objectives was valuabte and successful In each state department, 
however, a liaison had been identified by the ^ef as the primazy contact for 
activities of the CCSSO State Education Ay^misnt Center. Faolure to inform 
them of the initial contact to the mathematics specialists was a problem brought up 
by several of the liaisons, most of whom are state assessment directors. 

The problem, some assessment directon felt, was that mathematics curriculum 
specialists would not ahvays recognize some of the implicatioiis of the planning 
exercise-that these decisions would ultimately affectme state's results on the 
comparative assessment Assessment directon may be more sensitive to the 
political and educational ramifications of a high-stakes assessment than their 
curriculum specialist colleagues, because of their professional experience. 
Coordination between the two types of staff cannot be assumed, and it is 
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recommended that either everyone in any stakeholder orsanization receive much of 
the commwiication or that all communications go througn one designated liaison 
who is responsible for involving and informing the proper staff or committees. 

8) Work on the subject-matter objectives and on procediirnl and anfllytic plans should 
hecomc a staff function of the |ovemance process, and review by the field should 
become a part of the participation by thefield in the planning prnrgss 

^ 

The cooperation of people in the field throughout the consensus planning process 
was very positive, and it would seem that future planning efforts can assume that 
such cooperation will be present Funds were mtade available for states to 
reimburse e}q>enses of their review groups, but most states did not request funds. 
Members of planning committees served without conmensation for three, 2-day 
meetings. Those who worked on writing tasks beyondfthe three meetings were paid 
$2S0;^day, the standard NAEP consultant rate, but this work should be done by paid 
staff in toe future. This will be developed below. Token amounts were paid to 
scholars for mail review, again at the rate standard for NAEP, but this could be 
done without offering con^wnsation as the NAEP planning process becomes more 
visible and important to the field. 

9) The consensu* planning process should be "self-evahiaring". 

Several times during the plannmg project, committees noted that their decision 
about how to resolve an issue coiud and should be "t^ted". As recommendations 
are made for the assessment through the governance board or other consensus 
planning process committees serving it, procedures should be set up to evaluate how 
the recommendations tiim out Recommendations usually are intended to have 
some effect; the conunittee making the recommendations can often ss^ how and 
when one would know whether the intended effect has come about Tne 
committees should be asked to do this, to suggest how the recommendation could be 
tested, and staff should have a systematic melanism for foUowinE up on the 
e^^uat^on procedures that are suggested. This feature would add enormously to the 
continuity of the planning process and what is learned from it This is especially 
important when one recognizes that the govemanoe board will experience 
considerable turnover, asid that few if aiqr of its members will serve long enough to 
track the effects of individual po^des or recommendations through a complete 
assessment cyde. 

10) The planning nm^wM shmild have a built-in buffer to ensure that recommendations 
which are made are thoughtful and appropriate. 

A kwp could be built into each planning-committee activity, so that at least some of 
the more importam recommenoations made or polides set by committees or the 
NAOB are fold for some period of time until they can be considered again by the 
committee or BcMvd, before they become final policy. 

11) The Narional Center for Education Statistics must be adequately supported to staff 
and support the governance process. 

Adequate support means having aiq)rovals and go-aheads to su[>port the governance 
activity, having time to conduct planning and governance activities that are complete 
and sound, ima bavins enough appropnateprofessional and other staff to support 
the work of the new NationuAssessment Governing Board. These provisions must 
be made by Congress, the Department of Education, and the leadership of NCES. 
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National Assessment Planning Project attempted to approximate the planning 
activity that wU be needed for state-by-statc NAEP, using one full-time and two 
part-time sta£^ completing the work in eight months, and often awaiting poUcies and 
go-aheads while they were developed for the fint time. WhUe this was necessary in 
this instance, it revealed that more time is needed, that a larger and more 
comprehensive stafif is needed, and that the Department must establish and 
mamtam many policies to structure and support the process. 

ISSUES yfl) RECOMMENDATIONS IN THR nRnr CTVR.nKVirf nPMirvr 

1) An CTPlicit DQlicv^is needed hv the NAGB tn direct thn«> Hi^v^l o piny nhiecrives on 
y^J™!!? "^i*^^ Students do Imow and «.n Hn «nd ohiecrive. that ar^ 

^5;'^""*^"^ should to some deg;ee. he inHnd^d In'th: 
aSKSancnt if it gets too far outside what schnoh are currently teaching the 
WlUingnCiM to nartinnate will diminish. Thi% mnst y.. id^ from ahigh-fevel 
governance hr^ 

The primary issue confronted by this project's Mathematics Objectives Committee 
that wiL be at the forefront of future consensus planning activities was setting the 
balance between assessiiu what students jaaSL and saojlsi now and what the 
committee thought they should be taught This tension is present in any large-scale 
assesunent program: ghren that the results of the assessment wiU be interpreted to 
dnve ustructional improvement, should the assessment be gauged to lead 
msmiction toward a larger vision of what children should learn? Or, should it be 
desimed to isQssi what is cunently taught, so it can be a neutral gauge of how 
much of that intended or planned curriculum is learned? The former will lead 
teaching forward, the latter will tend to maintain the status gun. 

In the past, NAEP objectnws have been formulated to lead instruction somewhat, 
but to rest dose to current practice. In planning for state-by-state assessment, the 
resolution was set more toward leading the field, for several reasons. First, the 
states had already committed themselves, through CCSSO policy-setting on the 
issue, to strive for a comprehensive, forward-kxudng consensus and not to settle for 
a least common denominator of instructional content Second, the Mathematics 
Objecthfes CooLtiittee was favorably disposed toward the National Council of 
Teachers of M a t hema t ics Standards, which were intended as forward-looking goals 
for mathematiainttniction in the country over the next ten years. Hnall^^enthe 
ptpbkin of how to reoondle curriculum objectives that one state might address and 
o thaw ao ^ the Committee adopted the principle that the objectives should be 
in cnaiiiB , iBcotporatiiig those areas some states felt were important and not 
b gnwnig aliitoniyofwhatevetyoE^wasalreaihptirsuing. Ouce this inclusive 
approach was adopted, the compromise was made aroundoow much weight should 
be given to the various areas. Less weight was assig;Ded to areas that were more 
progresshre in nature or less prevalently addressed at this time. 

The scope of the objectives sbould be broadly inclusive of the curriculum and take 
into account the objecthres of the various states and districts. Hi^ should include 
what various icholan, practitioners, and interested citizens expect to find in die 
curricuhun. They shflud include the objectives diat were tested in tiie previous 
assessments so diat trend data can be provided (at least on tiiose objectives). When 
one voice says "a" is important but not "b" and anodier voice supports "b" but not "a", 
an assessment usually can and should include both. The cunent methodology of 
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matrix sampling of items allows for broad coverage, and analysis and reporting 
methods that link results to measures of opportuni^ to learn can reveal whether 
differential outcomes are related to whether students were exposed to V or to "b". 
Finally, a narrow assessment is the greatest and perhaps most justifiable reason for 
some educational leaders in a state to choose not to participate. They may stay out 
because their state's objectives are not included or because they believe that the 
assessment is attempting inappropriately to narrow or steer the curriculum through 
the influence of the test or both, but they will be less likely to stay out because the 
assessment represents a broader vision of what should be tested. 

2) The National Assessment Governing Board should have staff specialiste who can 
coordinate the meetinf; and draft material represenring the nhjective-development 
and other committees' work. 

The magnitude of the work that had to be done by the Mathematics Objectives 
Committee was underestimated in this project and did not account for staffing. An 
objective-setting committee must ultimate^ produce a report that is both 
suDstantive andsubstantiaL No person on the project staff was able to draft the 
report, so preparing it depended on committee members. This is a problem NAEP 
has addrettea by ««pg"«"g a staff spcdaM, to each subject-area assessment and 
structurins learning area committee meetings to permit the committee to produce 
wording^ chanses in existing objectives. In our case, the task called for a more 
basic consideration of the objectives, and this will be true in the future. A sub- 
conunittee was asked to prepare apreliminaiy draft of the objectives for the 
Committee, and this helped expedite the Committee's task. 

Staff specialists assigned to support the planning process in the future should 
include one q)ecialist for each subject that is assessed as well as others who are 
specialists in methodological issues. The assessment-development ^cialists should 
be trained and e>q)erienced professionals with either a background in the academic 
discipline being assessed and experience in developmg or managing instruction in 
that subject, or they should be specialists in the education of the subject with a 
demonstrated cq>«)ility to tap tninking of scholars in the disdpline as to its 
priorities. Duriiig each assessment development cyde (the two-year process of 
developing the objectives and exercises for a subject-area assessment) the person's 
priniaiy assignment should be supporting the development of objecth^es for their 
subject and ooordiiiating follow-up work on test and item specifications after 
objectives are devel<^M. 

3) netting pr^Hmlnaty €iigye«rinns from state department subject specialist and asking 
them well in advmcc to establis h state committees to review the draft 
r^«i«iiii»,nrfiirinn« aeemed pardoilarlv vnhiflhie in not only getting yood ideas, but in 
pliiriitg tlm M8cd for eventiia! support of thecontent of the state assessment 

Our effort to elidt state reviews of the draft mathematics objectives in February 
succeededpartly because each state was asked three months in advance to prepare 
for this. Tms gave them time to set up committees and a means for responding in 
their state that invohfed the appropriate people. This approach should be weighed 
thoughtfiilly in comparison with the NAEP <»perience of using a committee of 
seven representatives from the states for the 1988 assessments in histoiy and civics. 
Much may be gained by providing for comprehensive consideration and input and 
adequate lead-time to prepare for responses. 
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4) Systematic analysis of state curriculum frameworks shoul d be done and used ax 
background for setting objectives in the future, hut it shnnlri placed in relation to 
other resources and activities in the c onsensus-building prnre«. 

In the planning project, a contract with Education Products Information Exchange 
produced reports on the de^ee of alignment of state curriculum guides with the 
prior NAEP objectives. This was available for use by the Committee, but several 
factors limited the extent to which this information was used by the Committee. 
Fin,t, it was difficult to present the anafysis in a wav in which tne results were 
comprehensible and useful The jguides varied widely in their level of specificity and 
in the degree to which they constituted state poli^. Also, it was difficult to find the 
appropriate level at which to pitch the analysis. Guides varied widely in their 
content at a fine level of detail, but they were quite similar if one looked at their 
content at a more global level. The utility of this kind of analysis must be worked 
out through its details. 

Second, state committee review was ultimately important to the Objectives 
Committee as a check on the alignment and conqMuibility of state curriculum 
policies with the recommended objectives, as opposed to the content suoalysis. The 
content anaWsis should be redded more appropriMitly as a resource or reference 
when states begin consideration of a set of objectives rather than as a basis for 
determining the suitability of a set of objectives once th^ are drafted. 

Finally, the need to reconcile states' differences in their curricular policies was 
outweighed by support for a new, conq>rehensive, indusive framework for the 
assessment Systematic, objective conqMrison of the policies was sinq)ly 
overshadowed by the oonsensus-buildi^K phenomenon that occurred aroimd the 
prospect of a new set of directions. In tne future, the type of objective analysis of 
curriculum frameworks done by EPIE should be seen as useful background, but not 
as the paramount determiner of the conmion ground of the assessment 

5) A careful line must be tread between using the prio r objectives as a resource and 
aooroaching each objective-settiny e«erci«e as a ftii! and deep process of 
reconsidenny our values, needs and priorities in the subject. 

The prior NAEP objecthres were an inqxmant resource in the objective-setting 
process, helping structure the Committee's thinldng on how the objectives should be 
stated and presented, but the principle of basic reconsideration must le emphasized 
clearly to give the fovemanoe process legitima^ and to avoid inm>ropriate 
adhemioe to continuity with tiSe past 

6) F utmq flj rfMtiYMCttiny tailP should Indude a broad review of curricular issues, 
commiirioned at least eight months before it is need ed, and presented as a brief but 
Mih«f ftfjiM mmmafv tn h^ read bv co mmittee mem bers before their deliberations 
about the objectives. 

A ten- to fifteeiMMge summar|r of trends and issues seems appropriate and could be 

Erepared by stafif with appropnate participation by the field. A oroposed review of 
terature on mathematics curriculum priorities was conducted for ue planning 

groject in the form of a series of interviews with leading math educators, mostfy 
■om higher education. These interviews were summanzed and reported to the 
Mathematics Objectives Committee. The purpose was to identify issues and provide 
guidance to the committee so major issues were not overiooked. In this project, 
there was insufficient time to prepare a traditional literatiu'e review between 
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September and the Mathematics Objectives Committee work in early December. 
Several scholars were asked, but they declined to attempt the effort on the short 
time schedule. Interviews with experts in the field, reported orally to the 
Committee, were viewed by the Committee as a useml and adequate fall-back in 
this instance. But a longer time frame would allow a more traditional review to be 
prepared. 

7) Systematic solicitation of cnmment on nhjcctives from a broad rangft nf sc holars and 
educators should be continued and enhanced. The review should be broad, 
involving academic scholars in the discipline, specialists in its teaching and learning, 
teachers and schnnl fldminktratAfg anrf interested representatives of groups »uch 
employers. The review process should he hiyhly pihliriT ^. should be long enough 
to permit comment and should include procedures for expHcitlv incorporating 
review comments in subsequent drafting of objectiveg. 

In this project, comments were solicited by mail from scholars (math educators) on 
the prior NAEP objectives and on the NCTTM standards before the mathematics 
objectives were druted. This produced some responses, but more responses came 
in reaction to the actual draft objectives in January. All comments were 
incorporated by staff into the draft report conq)leted by the conmiittee in February. 

Respondents were paid $50.00 for reacting to the draft. As discussed below, this 
may not be necessary or ^propriate as participation in these activities becomes 
broader and is recognizedTby the field as a mors important event 

8) The NAEP planning process should become an event which drives consideration 
and development of national goals for education in a subject, through the process of 
helping determine what to test 

The NCTM draft mathematics curriculum standards became a very important 
document in the objective-setting process. It was the product of widespread 
involvement of matn educators over the prior two years. In the future, NAEP can 
capitalize on such developments if they are available in areas such as reading, 
wntin^ science, history, or dvics. If tney are not available, it may noi be feasible to 
commission the two-year process required to produce such standards as part of the 
objective-setting task in piamuBS an individuu assessment, but over time the NAEP 
consensus planning process will oegin to precipitate sudi efforts by professions, so 
they can be factoiea in to the consensus planning. There is evidence this is already 
taking place in antidpacion of the 1992 state-by-state assessment in reading. 

Again, consistent with the vision for the planning process discussed earlier, this 
event should become a very appropriate means for tapping the best thinking among 
our teachers, academicians, curriculum specialists, and the concerned public (such 
as the business community) about what in a subject should be taught and where 
instruction in that subject should be headed. Ultimately, the NAEP planning 
process should have oonsdously-developed links, estabushed by the NAGB and its 
staf^ between it and the analogues in other subject areas of the development of the 
NCTM standards. 

9) In fiinirc assessmenta. professional associationg and blue-rihhnn boards can be 
called-upon to guide thosg working on the objectives toward a range of thoughtfiil 
specialists, or toward eristing or ongoing efforts to define obfectives in a curncular 
area. The National Acaden^ of Educafion. the National Science Board, ana the 
specialist teachers' organizations could fulfill this role. 
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ImU^)^ the role of the Mathematical Sciences Education Board (MSEB) was not 
well defined in the planning process. They became very helpful in securing people 
for Committees and for mail review. f 

As part of the nrocess discussed ahovg. where setting the ohiective* fn- th<> 

assessment bccomM an event which predpitates thinirigg \^ thfi fifild . thf Ir i iiin rfifi 

COnununitV should be encouraged to ldenti^ their vali.e.s. need s, and prinriries for 
each subject. 

Busmess community input was difficult to achieve. Advice from others who had 
tned to invoWe busmess leaders in thinking about mathematics content was not 
encouraging. A literature review provided few detailed suggestions from this 
^pective on math content for schools. One recent report from the Committee on 
Educational Development recommended what content areas should be emphasized; 
It was provided to the committee as background informatioiL 

In future, the NAEP program rrm^\ jnclude an nr derk setpience of prodiiging 
Qbiectivei specifying t^t and item characteristiet. and d eveloping erercises. each 
clearW documented and available fnr review at appr opriate times. 

The point was made by some state assessment directors and mathematics specialists 
that m deciding whether to participate in a state-level NAEP, more detail would be 
needed on the content of the assessment than that provided by the traditional 
format of NAEP objectives booklet The Mathematics Objectives Committee 
responded to this bv including sample exercises in the report As the draft was 
circulated, the need for test descriptions and items became even more apparent 
than anticipated at the start of the project It was not feasible to produce these two 
documents in the time frame in which the project was conducted, and they were 
provided-for separately and after the fact by the NAEP grantee to support review by 
states of draft exercises. 

Producing these documents should be the responsibility of the National Assessment 
Governing Board. They should be conducted or contracted by the NAGB and 
conveyed to the Department of Education for use in guiding tne development of 
exercises by the NAEP operations contracts. The need for appropriate staff to 
suroort the objective-setting exercise was discussed earlier, the NAGB should have 
staff or contracting resources to permit it to produce test and item specifications, if 
i tdoetth is direct^, or to contract for them i! that is more convenient or 
appiepriate. Maintaining NAGB responsibility for objectives and specifications is 
the ifpropriate way to divide and execute responsibilities between NAGB, NCES 
and ttt NAEP q)erational contractor conducting the assessment as distinguished in 
the flfBiv legislation. 

The eonsenms pmtt^ .farting with esublishment nf the enmmitte g to develop the 
obiectivM apd to dev eliwi or overaee the development of item mcdfigatinns should 

SUfficienthr in advance of suhmittiny the report* to NCESTfor conveyance to 
the NAEP operational contractor. 

The original time-line for developing the math objectives in this project was 
extended when it became apffutnt uat the work-plan for January and February was 
not possible, because of the time reouired for various reviews. Tne completion date 
for the reconunendations was extended to April 1, 19813, when NAEP staff reported 
they needed to start drafting exercises by that date. Even still, seven months from 



SVlj?§ong°en«dfy "^^^ ^^"^^ thinking from the field and 

If item writing for the 1992 assessment must start April 1, 1990 the followintt 
sc ledule seems to be a minimum: ^ 

NAGB establish conunittee, prepare 
for its start, alert field 

to process~2 months Dec, jggs 

Start development of objectives 

by committee--6 months Feb. 1, 1989 

Start development of item 

specifications, 5 1/2 months Aug. 1, 1989 

To ED for review~ 

Two months jan. 15, 1990 

Revision by NAGB and conveyance by 
NCES to operations contractor to 

begin preparing exercises March 15, 1990 

For the planning of a state-level assessment according to the approach used in this 

the bcgmmng of item wntmg by the NAEPopcrational contractor" TOsS to 
provide six complete months after the establishment of the Objectives Committee to 
develop objectives, after the committee is organized, and it provides more than five 
raontta for the development of test and item specifications by staff or a contractor, 
afifil the objectives are complete, but before test-development begins. If the process 
IS even broader and more time-consuming than the one reported here, which it 
should be m order to include more tapping of thinking in &e field, then that time 
should be extended at least su months. 

13) Over time, what each succeMiw. «M^Mm^nt mMmrcx in « !,uhiect should h<> 

fmmiQ,Tfi\mm9 rhanffet ir what nrarritionew^ ^hnlar* and int^»Kt..H ^tiy^ 

yiifflgfe^id be taught. ofh;;c:i^"Vh".;v;:;c^^^^ dis::;!.;;:'^;^^^ 

llljhejgagd thy nwtifv rhang^ in onr v>ur nf what i"?mPfan n^^^ ^Z! 
ana ne ahie to do. 

Haviof trend data to measure progress based on some of the same objectives 
between assessments is important It is not likely, however, that subject matter 
content would change so much over two to six years (between assessments) that 
there would not be enough common objectives to provide trend data. Analysis and 
'*^d?°* "®"°**°^°8y can differentiate success with new objectives from historical 
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gj^g.??!?:^^™^'^ PLANNlNr, PROrFni TRES. AMAf YSfS, AjSip 
KEPQRTINCf 

0 Planning nrocedural and desip featur^^ should he rfnne under a snirit of basic 
consideration of the issues. ^ 

As in the Math Objectives Committee, this group made many recommendations that 
departed from past NAEP practice. They were advised at the outset not to be 
umited as a constramt to current procedures or resources. For example, they 
recommended reporting assessment results by curriculum sub-area rather than just 
by the overall NAEP proficiency scales for the subject This freedom of scope is 
essential to a legitimate consensus pl anning process. 

2) DevelCDment of procedural and analytic recommenHarin ns should he done hy a 
consensus planning comimttee <upported hv appropriately gualifierf ttafp 

In the planning project, it was presumed that committee recommendations c uld be 

Ercpared at least in preliminary form by the Committee. This was impossible; staff 
ad to work from committee deliberations and decisions to prepare a report 
containing recommendations. This will continue to be true m tne future, and the 
NAGB should support procedural and analytic planning committees with staff. 
These sta£f should nave expertise in design issues induoing mmpiing^ scaling, and 
analytical models; they should also have expertise in practical matten in the 
administration of an assessment, including ensuring uat procedures are valid and 
consistent During each, two-year assessment cyde, NAGB should have one full- 
time staff member whose primary responsiblity is to help planning committees seek 
mput and advice on these issues, prepare recommendations and other reports, and 
work with NCES and the operational contractor on the implementation of 
recommendations. 

3) When changes are recommended in the NAEP design or procedures, thev should he 
evaluated and thev should be controlled to avoid making ton m any changes at once. 

While the procedural features for the state-lsvel NAEP were being considered, two 
spt'dfic problems brought this issue to the fore. One was exploration of possible 
wuses for anomalous results in the 1988 reading assessment The other was 
consideration of how to stroctiue state and federal responsibilities in the expanded 
NAEP-whether to change the mode of test-administndon from contractor 
administr ation to cooperative administration by state and local staffl Tbe PAR 
Qnudttee felt, that in these instances lay a strong rationale for introducing new 
proeedoral changes onfy under conditions where their effects could be controlled 
and measured. 



3) WMlfl me issues conftnnted hv the PAR rommittee seemed fairly technical and 

specific, and their resohmon bv the nommittce m«y hmve seemed c ut-and-dried. the 
ISaiea were in many ipanv ways as profound as the determination of nihject matter 
which was confronted bv the Mathematics Qhiectives Committee: 

Determination of how state-by-state results are to be presented, tbe degree to which 
background conditions are to be taken into account, and the educationil processes 
with which achievement results are to bs related at the state level are weighty tasks, 
philosophically and educationally. Tlieir resolutions, while impearing to constitijte 
straightforward methodological decisions, have great political and educational 
import This fact should be remembered as these issues are considered, so that their 

ERIC 

20 



cnt^sideration is not erroneously relegated to a mundane methodological realm, 
b jause these issues will refuse to remain in such realm. Their soci^and political 
nature must be confronted squarely and by people with appropriate insights and 
expertise to resolve theoL The process also revealed that a plaiming committee can 
successfully lay down design principles and decisions that can guide that project. 
These deasions need not be left entirely to staff to resolve along the way; many of 
them must be resolved by a politically-representative body. 

4) NAEP should have a modest pool of resources that are devo ted to naw idea£. while 
the practical const raints and need for caution in the ongoing program are honored. 

The tension between the assessment being descriptive and being prescriptive applies 
to the consideration of technical issues as well as to consideration of content. Here, 
the issue for the future is the effect on innovation. 0>nsensus plannii^ tends to be 
levelling, toward a common denominator ofprofessional opinion. This is 
exacerbated in the instance of plannii^ NAEP, because of the constant need to plan 
an assessment that must be in ue €ela in twj or three years. Although very bright 
ideas may emerge in the planning process, if they are too innovative they are 
discourased or shelved in light of the practical need to olan sometlung that can be 
done ana that will be techmcally reliable in th<* near term. Unlike innovations or 
progressive stands taken on the content t o be assessed, innovations in 
memodological areas often involve real risks and material costs. Progressive 
content, m the other hand, risks only faUing short and not progressing as quickly as 
we desired. These risks are non-specific and the cost of bemg ambitious is low and 
distant. Therefore, there is a need for the planning process to be complemented 
with a means to support methodological innovation. 

No more than $200-250,000 would be required to support as many as a dozen 
specialists working on the viabib'ty of their innovations. Such a program could be 
conducted by NAGB or, under its direction, by NCES, as a discretionary program of 
grants awarded competitively to contributing planners, analysts, and technical 
specialists in the field. 

5) Thfe most inte llectiifllly demanding issues addressed bv this committee were how to 
handle demog ranhie data in reporting results and how to relate results to 
characteristia of schn. J profranw. A consultant paper on the issues was useful in 
guiding the rnmmittee toward ia recnmmendatinns, fc the future, this issue and 
T.OW tn handle it terfinicalW will continue to demand thinkinghv Planning 
committees and further dev eloptnent of techniques A NAvtH siihcommittee. 
a.-med with contemporary consultant papers, should be dedicated to this and related 



Abo, since the expanded NAEP has as one purpose more informed iwlicymakin^ 
the presenution of results should be worked out through a back-ana-forth plannmg 
process involving representative policymakers and planning committees on the 
desim of the formats for presentmg results. Ideally, this should be done around 
mocK-ups of possible displays. 

6) States and other const^ U^ncies should be consulted in the future on procedural and 
design issues. 

Input from states on PAR issues was timely and valuable. Fully half of the states 
either made comments at the outset or responded to drafts on the PAR issues. 
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States contributed substantially to the resolution of some technical issues One 
state, for example, provided a simulation of how to use demographic variables to 
mterpret states' comparative assessment results. This should be solicited and 
encoi^cd la the future. Most states seemed more interested in gettine adequate 
specification of siimfi solunon, rather than being interested in one particular 



7) Professional associations nrnvided uk^M rm^uTce& on snmi. issues; thi.ir h<>ln 

should be sought on snerific icci.. in pi^n ni;; in th;"ft;t i';;ft,''^"^"' '"^'^ "^"^ 

The Question of hpw to spedfy rules for inclusion of special students was solved 
largely by the National Assocation of State Directors of Special Education, and the 
Association of State Assessment Programs provided very concrete advice on the 
development of test specifications. Tliese resources can be relied-upon in the 
future, and others can be brought in to help on other issues. 

8) The development of item snecifi<ations must be hniit inm thi » planning prnri»« 
DMcede Item wnting. This should r^«.it in ^ fTeeSZyH)^'^^^^^^^ ^" 
addinon to the nhiective booklet and that c^n h^'i^i^^H h^r srSirill^s'nnnr 

development of eTercises. --™»jM...u«,««ajua4*uyi^ 
CONCLUSION 

th* f^^SrS,^^^ ^jscssment Planning Project served two purposes. One was to make 
the transition substanttvely from a National Assessment of Educational Progress to a state- 
by-state one, to actually decide how that would be done. The other was to serve as an 
experiment or stalkmg-horse in the planning of something as sensitive as a state-bv-state 
educational assessment in the United States. This report is aimed at realizing that 
expenment If the proiect was successfiil, the observations and recommendaSons put forth 
here will provide the National Assessment Governing Board and its staff and custodians in 
tlie Nanonal Center for Education Statistics with something valuable, based on this uniaue 
experience. ^ 
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development 
process 
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efidrts of all the indMdiiais %i4io GOf^^ 
to the development of a reading assess- 
ment. Many people* including univcrsify 
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84 and the 1985^ reading as^ssments— and extends 
at least through the 1987-88 reading assessment. 

Special thanks are due to the memt>ers of these com- 
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for the assessment %^re responsive to the revie%^. arul 
spent long hours reviei^ing and revising ol>|ectives and 
exercises. Appreciation is also due to Kalle Qerritz. read- 
ing coordinator for MACrs 1985-86 assessment. 
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Introduction 
Ot overview 

The Development Process 

ince 19G9. The Malion's Report Card, the 
national Assessment of Educational Pf o- 
gress (flACP), has been responsit>le for col- 
lecting information alXHit the educational 
achievement of our nation's youth in a vari- 
ety of sul>iect areas. These areas have included reading, 
mathematics, writing, science. music« art. literature, com- 
puter competence, social studies, and citizenship. Civics 
was first measured in the context of tt\e 1969-70 Citizen* 
ship assessment, again in the 1974-75 Citizenship 
assessment, and a third time in 1981-82 as part of a 
combined assessment of citizenship and social studies. 
The otijeclives contained in this t>ooKlet represent a mod- 
ification of some of the 1981-82 Cilizenship objectives 
and additional ot>jectiv€s which reflect current trends in 
cMcs education.* 

The new statement was developed by HA£P's teaming 
Area and Slate Advisoiy Committees and has been 
reviei^ extensively bjf teachers, curriculum specialists, 
and school administrators to ensure that the assessment 
topics do noC diverge sut>stantially from current instruc- 
tional practice cr expectations. All of the contributors and 
reviewers %Mere selected to reflect the perspectives of peo- 

CS- 101. OciMcr CO. MailofMl AsMsi&mcnl ol tduc^UucMl rtoj^csi. tUt^cjiiufi 
CommiuiDn ol Che SUlcs 
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pie in various sizes and types ol communities, from many 
geographic regions, and from a variety of racial/ethnic 
groups. 

This final statement does rH>t necessarily reflect the 
views of every individual %^ participated in the develop- 
ment and review processes, but presents* as accurately as 
posslt>le, the consensus reached. 

The Assessir ent of Civics: 

United SUt .s Govcrnacnt and Politics 

The assessment of civics lake plaot during the Bicen- 
tennial of the %i^Ung of the ;^>nlted States Constitution, 
the oldest written national consUtutlon In the world. Par- 
ticipation in the government ol the United States by aU 
Individuals is %4iat forms a democracy —our government 
is of, by. arMl for ttie people. To endure, a demoaacy 
relies on the alMlity and %^llingness of its citliens to be 
involved intelligently in political and public affairs at all 
levels: local, state, national, and international. 

The tMToad purpose of including U.S. government arul 
civics in school curricula is to prepare students to reflect 
on and participate in the political dedslon-maldng pro- 
cesses of our society. Understanding democratic princi- 
ples will help students appreciate and exercise their rights 
as well as itrcogniie the responsit>ilities inhe^rent in l>eif g 
a United States citiien. Studying the structure of our gov- 
ernment, the functions of its three branches, and the 
politUial processes by %^ich decisions are made will 
enat>le students to participate more fully and effectively as 
informed dtiiens. 

Tiie Objectives Frameworii for Civics: 
United States Govcrnmenlt and Politics 

A three-dimensional inalrix made up of Content. Context, 
and Cognition outlines the broad objectives for civics edu- 
cation Figure 1 presents the in.itiix and shows how each 



dimension is divided into major categories. tLach exercise 
in the assessment can t>e classified wit>iiii ihe m<itrix by 
nialchiny the category of content it assesses, the context 
in which the question is asked, and the cognitive skill it 
measures. 

Together, these dimensions help di'tine the major 
objectives of United States civics education and provide 
guidelines for developing qu'^stions to assess students at 
age 9/yrade 4. aye 13/yrade 8, «..«d a(|e 17/yrade 12. 
This concepluaS framework and the objectives themselves 
may Im! useful to cuiriculum developers in their, own con- 
sidei Uions of scope and priorities at state and Jocal 
levels. 
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IRirticipants 

in tne 
development 
process 




he national Assessment appreciates Ihe 
efforts of all of ttie individuals %i4io con- 
irilMJied to the developtneni of the 1988 
Civics Ot>fecttves. Many educators, includ- 
ing unh^rsity proic. ors. histoiy and social 
science researchers, classroom teachers, school adminis- 
trators, and curriculum specialists, as well as concerned 
parents and lay persons, participctted in developing and 
revie%^ng successive drafts. These ot>iectives could not 
have t>een developed without their substantial involve- 
menl. The national Assessment wishes to extend its grali- 
tude to all participants. 
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C 0alr Greenwald, learning Improvement Services. 

riederland. CO 
David Harris, Oakland County Public Scliools. Ponliac. Ml 
JoAnn tieidenreich. Millwood High School, nashville. Tn 

SyMa Hoffert, St. Louis Country Day School. 
St LoMis. MO 

Heverend Giles K Nayc^. O b IS . The IXtlbailon ScIuk)I. 
Miifii^Uiwii. hJ 




Linda Giickstein. Pliiladelphia. PA 
David Glickstein, Philadelphia. PA 
Paul Dennis noffman. Canyon dc- Oro High School, 
Tucson. AZ 

Donna Hudson, Old Center Elementary Scliool. 
Antkxih, Tfl 

Alison Johnson. Qlencliff High School, ndshville. Tfl 
Savannah Jones, Birmingham Public Schools. 

Birmingham. AL 
Syhfia Hamowsky, Overton High School, nashville. Tn 
Joseph Hovacs. Edison School District. Edison, nj 

ikivid Laudenschlager. Kapid City Central High School. 
Rapid City. SD 




James Loguidicc. Bucks County Intermediate Unit. 

Ooyles^oiMi, PA 
ThooBSS Lyons, Phillips /icademy, Andover, MA 
Anne HdUk. Kirkpalrick eJemcntdiy School, 

MashvUle. TN 
Maiy HcTmrtmHi. Parkway School OislricI, 

aicslcrfiekl.no 
OaHdMelsm^. Portland. OR 
ray micmlt. MalkNuM Council of the Social Studies, 

Washington. DC 
5. ilev Manvw, Pienn^lvanla Slate Univeraify, 

University Park. PA 
Csrim fUnkeivIs, Hcigs Naand School, llashvWc, TTI 
Louise Osborne, ralMtamlHon Clementafy School. 

flashvillc, Tfl 

Aithur Ftsse, Letianon High School. Lebanon. Iltl 
rtederick W. Hbler. BkXMnAekl niNs. Ml 
Dougtss FhUUpA. Anchorage Scliool District 

Anchorage. Aft 
John FhiiUps, California Dcpartinenl of CducatkNi, 

Sacramento. CA 
CMhertnenckie, Memphis City Schools. Memphis, Tfl 
Gene FIckeL Oak fUdge School. Oak Rklge. Tfl 
helen Mchsrdmm, PUHon Cqunly Board of Education. 

Atlanta. OA 
Susan Aolierts. Albuquerque NM 
l4Kille/lo6^ntsoa Ontarto-Muntciair, Ontario. CA 

AiRoccM, Sequoia Junfor High School, Redding, CA 
Jkxto llMS/. CI Cerrtto Scnfor High, El Cerrito, CA 
Jennie Scott Moore Mkklle School, nashville, TM 
Denny ShliUngs. Homewood^riossmore Community High 
Sctiool, nossmorc. IL 
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LmnySUricki nd. Public InstrucUon Office. Tuniwaier. WA 
Ruth Sykes. haywood Ctementaiy School, llashville. Tn 
amine M. Takenak^, Department of Education. 

Honolulu, Hi 
Jan Talbot. Pair Oaks, CA 
Steven Teal Hercules, CA 

Alan Teeter. Percy Priest Elementary School, nashville. TM 
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Fml Tke, licliurray Middle School, flashville, TH 
MaiyJmne Jtumer, Center for Ovic Education^ 

Calabasas. CA 
Jane VMmlercook, hunters Lane Comprehensive High 

School, Hendersonvllle. TM 
Deborsh Welch. ATIA. WasWnf^, DC 
Joy Wekkm. Bemiehem. TA 
Maty Lou WUIIams. SanU re. m 
Thomssine Wihon, Berfceky, CA 
Wrg/nte WUson. nc School of Science and MalheniaUcs. 

Durham, flC 

Celesle Woodley. Boulder Valley Schools, Boulder, CO 
Brian §L Wright lleely's Bend fttddlc School, 
nashville, TM 
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PUBLIC LAW 98-511, SECTION 405(e) 
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PUBLR LAW «8.SII. StC I ION 405(c) 

Oi Im addition to the othtr rmaomnbilitiMt of tk^ ntn^ j 

cdulto iM the basic ekilb^ ^SSSUUtt^ ^ 
•ad other euhjocte oad Mlh. flMi • iSSI2r2L2r'~?*?'i?* 
lAt collect aad nport aTCJ^T-Sfl. 

a. (S^XSZi!^^ 

(D) utchtde m ntummtnt metiuSmiSSZM^T'^ 
groum ofiadividtimk: imformimtiom am special 

^•t^of^tobmetitm. primarily^ertaimiMM tol ^ 

hihTXiairss^^ 

«»om. 1^A»memmtJ^dic,0ammiUa^^ 

cauffonn of mtcmbenhip apacifiod im tubpangnph (BX 




ERIC 



(B) Memben of ihe A$ae$$meni Micr Committee appointed in 
aeeonlanee wiih eubparagmph (AXiO •hall be^ 
*rO one ehiefS$aU eekool officer; 

two Staie kgielaion; 
*Viii) two eehoot dietritt euperintendenie; 

(iv) one memkber ot a State board of education; 

(v) one member ^ a local echool board; 

(vi) one Governor of a State; 

(vii) four elanroom teacken; 

(viii) one elementary echool principal; and 

(ix) one $eeondary echool pnncipaX 

(O Hie Ae$i$tant Secretary Aall serve a» an ex officio member 
of the Aeeeeemeni Policy Committee The Auistant Secretary shall 
aUo appoint a member of ihe Council to eerve a$ nonvo ti ng member 
of the Aeeeununt Policy Committee 

(D) Members appointed in accordance with eubparagraph (A) (if 
and (%i) ehall be appointed for terms for three yfars on a staggered 



(SJ The Assessment Policy Committee established by paragraph 
(t) shall be responsible for the design of the National Assessment^ 
including the selection of the learning areas to be asses sed, the de- 
velopment and selection of goal statements and assessment items, 
the assessment methodology, the form and content of the reporting 
and dissemination of assessmeni results, and studies to evaluate 
and improve ihe form and utilisation of ihe National Assessment 
The appropriateness of all CQfniiive, background, and attitude items 
developed as pari ofth^ Natwnal Assessment shall be ihe rssponst- 
bility of ihe Assessment Policy Committee. Such items shall be sub- 
ject to review by the Department of Education and ihe Office of 
Management and Budget for a singk period of not more than eixty 
days. 

(4) Each learning area assessmwnt shall have goal staUments de- 
vised through a national o'^nsensus approach, providing for active 
pariicipaiion of teachers, curriculum epecialisis, sukieet mustier epe* 
cialists, local school administrators, parents, and members of the 
general public All Hems selected for use in the assessment shall be 
reviewed io exclude Hems which uUght reflect racial, eex, cultural, 

^ '^llhSrt^paiiom in ihe National Asseument by State and local 
education agencies mlected as pari of a sample of euch agencies 
sholl be vtfluntary 

(t) The Secretary shall provide for a periodie review of the Na- 
Oomal AsssesmenL This review shall provide an ^oriuniiy for 
public commeni on ihe conduct and usefiilneu of ihe Naiional As- 
sessmfnf and shall rssuli in a report to ihe Confess, ihe Ptmideni, 
and ihe Nation on ihe findings and recommendations, if any, of ihe 
review The Secretory shall consider ihe findings and reoommenda- 
Hons in designing ihe competition to select the organixation through 
which iheOfTwe carries out the National AesessmenL 




APPENDK D 
PUBUC LAW 100-297, SECTION 3403 



102 STAT. 344 



FUBUC LAW 100-297-APR. 28, 1988 
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SEC 3241. DCF|NrnON& 

For the mirpoie of this part— 

(1) the term "at risk" means students who, because of learn 
in| defideocies. ><«ck of school readiness, limited English pro. 
fioency. povertjf, educatkmal or economic disadvantage, or 
phvakal or emouonal handicapping conditions face greater risk 
of low educational acliMvement and have greater potential of 
beooniiig school dropouls; 

(2) tha^ term "Boud** means the Fund Board established 
under section S2I1; 

(3) the term "Pimi** mmm the Fund for the Improvement 
and Referm of flchasis and IWching established under section 
3202; and 

(4) the term ''Sacrstaiy* oseana the Secretary of Education. 

PART C--NAT10NAL AHMflHMKNT OF EDUCATIONAL 
PBOGBIBB 

8BC MSI. SHOKTmUL 

This part may be dted as the "National Assessment of edu- 
cational Pirogrsss Improvement Act'*. 

UC l4Si. AT AliaiINT or PUBPOM. 

The purpoas of this part is to improve the effectiveneas of oui 
Nation'a schools by making olijectivw informatkm about student 
performance in ^iJioei^bd learaiiw arsae nvailable to policynmkers at 
the national, ngkmal. States and keal ^'^vaia. To enhance its utility, 
such informatkm shall be both nwsasntativa and comparable and 
shall be maintained in a nMmner that ansursa the priva^ of individ- 
ual atudanta and thsir faasilisa. It is not the mirpose of this Act to 
arthoriaa the cnUeciion w rsportiiw of iniormatkw on student 
attitudaa or bslisfc or on other mattan that are not germane to 
the acquisition and anatyva of infonnalkm about t tfi i 4f " ^^c achieve 



r OP BDUCAnoNAL raoGBua. 

(a) Obnbuu. Aiimoninr.— Sectkm 406 of the General Bducatio.. 
Provisk>MAct(haroaiUrinthiapart i^r adtoaa'*<heAct"andss 
amended by asetkm 8001 of this title) is amended by rsdeaignating 
stibascthm (i) as aubasctton (|) and by inssrtfa« after subaectkm (h) 
the follosriiuc 

*%W} m& the advke of the Natkmal Asssssment Goveminr 
Board eatsMishad hf parHnph (SKaKU, the Oommisskmer ehsll 
carry out. by grants^ contracia, or oooperative i^paements with 
quaflfled otvan&ntkma* or consortia thersof; a Natkmal Asssssment 
of Educatkmal PMfrsss. The Natkmal Asssssment of Educatkmal 
shall ba^idaosd in the Natkmal Osnter for Educatkm 
1 shaU rsport dirsctly to the Commisskmer for Bdu- 
itios. The purpose of the Natkmal Assessment is the 

It of the performance of children and aduha in the basic 

skills of rsading, nmthamaties. aclnoa» writing, history/geography 
and other areas selected by the Boa» 1 

"(2XA) The Natkmal Asssssment ^*haU provkle a fair and accurals 
prsaentatkm of educatkmal achievement ui skills, abilities, •na 
knowledge in rsadkig, writing, nmthenmtka, sckmce, history/gsog* 
raphy, and other arsas specified by the Board, and shall use sam- 
pling tachnkiusa that produce data tlmt are roprseentative on i 
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PUBLIC LAW 100-297- APR. 28. )988 



"liSieW^sSiSr"' ""■ """^ •"<"«• «iM 

-■SSSS^llHliS' 'TO^x*' «»^«m»niiig whether .uch an 

iJSLm^llku "SSLU'T.** PMtidwite in the demonstration 
oI!£Sl^ilJ£!S!ftf^ knowledge of the procei foj 

SS^t!^ 7SST2"5' y ot»cU^ to be tested, required in 
SESSf tI2Sj5Sj'^'T!!!f"* demonstrrtion ^timdartta fS 
StSS' 2i5te"flf^**^^*?? "^"^y- «»"«ction. valida- 
S!!^'^ntTiSSS\l^ ■'r^*"' P«rti«l*te.«h«ll sign an 
^l^S^LSVS^ the .Commissioner. A participating Stat.' 
5S of from uy test 

S^^^STi? JSk aS? Rtf" E!^ ^^l"* demonstration pAoVH 
«I data. ReAisai liy a SUte to re ease its data ;i 

t2''!J!i^JSST!'^' ••l?" P"^'*^ f^' independent evalua- 
nSSj ^KiS.'^ftT''^ o^aniaatlon (such as the 

SSSS^it?^' *^ ^ National Academy of Edu- 

22Jii2r.fl?Kf7^-" feasibility and validity of 

•jMsmento and the fairness and accuracy of the data they prtxluce 

IS "l^ "i^" ^ technical problems eKSSS 

•nd a descnotion about wk.» u..^^ .kT... 1.." •.r?"^'^ 



102 STAT. 34 



HeporU 



ContracU 



Report! 



» the National Assessment of Educational Progress. 



««»niBni 01 uiucational Progress. Th-? 
jwlts of this report will be provided to the Congress anJ to Stated 

S)S5tC laSSh^'f^tr?*"'' ^ paragraph (C) (i) aS 

"fni-» ^ 2*?**" f}^ aasessmenU were condui yi. 

^)S^'^J^ A-jessment shall have the authority to de- 

J*P ^P,™*"**! "Pon th? direction of the Board and suW to 
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02 STAT. 346 PUBUC LAW 10a-297-APR. 28. 1988 

"(3XA) The NaUonal A«e«iMnt shall not collect any daU that 
are not diracUy related to the ap|N«iad of educational performance 
achMvementa, and traditional domofraphic reporting variables, or 
^'^.^v^jy •ccuraU pw wi U a l i u n ef such information. 

'm The Natioiial A ■■■■ m i n t ihaU provide technical aMiatonce 
to Slataa. lociditiaa. and other puitias that deaire to participate in 
the ■■iw n i n it to yisU odditMiial iaJbrmation dsacrifaed ihpai^ 
graph (2). 

"iHM Buopt as pravidad 10 subporMraph (B\ the public shall 
ham aooMs to aU data. fimHoni. and tast instrumsnts of the 
National Assesmsnt. 

*ma} The Commissionsr Ml smmio that aU personally identifi- 
able ufiKiution aboue alaiMii^ Ikrir adii^^ 
their bmUias and that ' with resprsA to individual 

schools rw»ain flonfMsntfal, hi wwdaiMn with ssction 5S2a of Mtle 
6.Unitod8tatssCMs. 



"in) NotwithsTandiim aaj othir novWon of the law. the Sec- 
rotaiy mm dadino to imJw bvaikbla to the publie for a period not 
to o»MdT0 ]M fBllowii« tlMir lytU M oi«aiti 
the Secretary intemb to Touss in the ftitura. 

"(O The use of NnttoMd Oainiiiit tast ftams and test data 
employod Ji the pilot pngrani authoriasd in subssctkm (2XC) to 
rank, oompare, or otherwise evaluate individual students, ■choob, or 
school dMtrictoteBNhiUtad. 

'ISMAKi) Thofa h satahlMiad the National AsHSsment Oovemint 
BoanKhanaftortotUaaacliaanforrsdtoasthalioani'). 

"iU) Ths Boaid abaU famulato the policy guid^linss for the 
National/ 



"(B) Ths Board shall bo apiMintad by the Secrstaiy in aoooidanoe 
with thia suhparapnph anir aubparagraphs iO, (O). and (E). Ths 
Board shall bo compossd of— 

"(i) two Oov ra or s , or former Governor*, who shall not be 
members of the same political party. 

"(ii) two State logislatorH, who shaU not be members of the 
ssmo L^itifal pai^ 

"Uii)hro chisf State school officers; 

"(iv) one suporintandsnt of r local sducational agency; 

"(v) one msariisr of a State board of education; 

"(vi) one maadmr of a local boaid of education; 

"(vii) thrso daasraom taachsrs rspnssnting the grade leveb 
at which the National AsssHmsnt is conducted; 

"(viii) one rcprsosntativo of buainsss or indu^ry; 

"(is) two curriculum spscialists; 

"(a) two testing and msasuremeni experts; 

"(si) one nonpublic school administrator or policymaker; 

"(aii) two school principals, one elementary and one second- 
ary; 

'(xiii) three additkmal members who are repreeentatives of 



the general public, including parents. 
The Assistant Secretary for Educational Reeearch and Improvement 
•hall If rve aa an ea officio member </ the Boerd as a nonvoting 
member. 

"(CXi) The Secretary and the Board shall ensure at all times that 
c:: ' the membership of the Board reflects regional, racial, gender and 
^' Y cultural balance and diversity and that it exercises its independsnt 
r-ni/^ judgment, free from inappropriate influenoea and special interests- 
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"(ii) In the exercise of ite functions, powera, and duties, the Board 
'^^ independent of the Secretary 

•^i • »%2''oL"*!''*' *>f ^ Department of Education 

(III) The Secretary may appoint, at the direction of the Board, for 

^'ufl;?^ Sff^J/*""* t*» proviaionii of title 

5. United btates Cade, governing appointmenU in the competitive 
rrvice,. not more than 6 technical employees to administer this 
•ubsectimi who may be paid without regard to the provisions of 
chapter 61 and subchapter HI of chapter S3 of such title relating to 
clswncation and General Schedule pay rates 

"(DM The members of the Assessment Policy Committee, serving 
onthedate of enactment of the National Assessment of Educationid 
Progress Improvement Act. shall become members of the Board for 
the remainder of the terma of the appointment to the Assessment 
Policy Committee. 

"(ii) To complete the initial membership of the Board, the Sec- 
retaiy shall apmint members of the Board as necessary in the 
cstegsnei described in subparagraph (B) for which there are no 
members continuing from the Assessment Ptolicy Committee on the 
date of enactment of the National Assessment of Educational 
Progrsss Improvement Act The Secretary shall appoint such mem- 
bersfnm am furnished by the Governors, chief State 

Kliool oiTioerB. eduction associations and oiganisations, the Na- 
tional Academy of ScieiiMO. the National Academy of Education, 
pamt organisations, and learned H 

"(III) As vaMndos occur, new membera of the Board shall be 
appointed by ^ Secretary from among individuals who are nomi- 
nated 7 the Board after oonaultation with representatives of the 
croup listed in aubDara|raph (B). For each vacamy the Board shall 
nominate at least S individuals who. by reason of experience or 
^ *^ particular Boord vacancy. 
m Dtembers of the Board appointed in accordance with this 
psrsgraph shall serve for terma not to exceed 4 years which shall be 
itegBered. aa dewrmuied by the Secretary, subject to the provisions 
of subparagraph (DKi). Any appointed member of the Board who 
changsa status under subparagraph (B) during the term of the 
appointment of the member may continue to serve as a member 
w>Ul the expiration of that turn. 

D out «*■ functions under this subsection, the 

Bosrd shall be rssponsible for— 

"(i) selecting subject areas to be assessed (consistent Kith 
para^ph(2)(A)); 

"(ii) identifying appropriate ach^vement goals for each age 
and grade in each subject area to be tested under the National 
As sess m ent; 
"(iii) developing assessment objectives; 

"(iv) developing test specirications; ^- • 

"( v) designing the methodology of the assessment; < > t 

"(vi) developing guidelines and standards for analysis plans 
apd for reporting and disseminating results; 

"(vii) developing standards and procedures for interstate, re- 
gional and national comparisons; and 

"(viii) taking appropriate actions needed to improve the form 
and use of the National Assessment. 
"(B) The Board nuiy delegate any functions described in subpara- 
|raph(A)toitestair. 



"lO Th* Board alwil iiiave fiiuii auUiority oo the appropruteiM^ 
Of oqtnitive itmm. 

tJ!^S!J^JS^ f"*^' ■tepa to eiiMire that aU itona aalact*! 

torvm in tha Natwnal h m m it ara fraa fram racial. cultSJ 

•MMtor. or ranooal biaa. 

nJJrLiSf* aawamant ahaU have foal atataiMni. 

tSiitel*^,' "ftiMial coiiaaMua appnMch. pwrndiaJT^'* 




oonMltula iMd nrntimu mMmktA by thaKd) M^taei?^.; 
th. h«cti«» ^ 

'^B) Plor tha j 111111— «riliaSfaiMiath» AiDctioiML tha Ba..u 
•|»U ha*, tha MlhStiM gSSSl^^ 



of that law. 

•KB) Plartkipatini hi ■iiiiiwiim SaWSShaaia ahaU k 
S?*V"teJ^?i!?^ Mwamant with an* 

- "" ■ ^ tha State undj 

thia auhMCtioB. BMh audi ivraaMOt ah«U csotahi prov^ 

tor~* 



'H^l^ ^ ^ IMwticipate in the aMMment; 
J^LVlS^ the State wUfpay from noo-Padaral aouitea the oon 
Federal ahara of partki|NMion: md 

JSSL??^ 22iff?».«W with the tora- and oo»ditiora 

KOuidr Mch fiacal year, tha ooolPMeral ahara for tha purpoae 
ofdMBa (U) oTaubMnfruh (B) ahall ha the coat of conductiiiff the 
iiiiM nm it hi the State bKludiiw tha coat of adminMlannf the 
1 1 ■ > n il 1 1 n t at the achoel leral fcrJl achooh hi the StateMmg 
the ooatofooonUnation within the SUto. 

"(ii) Ttm MB-Pladeral ahara of iiayiiientB under thia pan«raph 
may bo in caah or in kind. 

.J^W- <?«— ' W *W M r ahali provide for oonUnuing leviewa of 
the NatMNMl AaMMMot, indudiim validation atudiieVy the Na 
tiooal Canter for Education Stetiatioa and aolidtetion of public 
comment on the conduct and ueefiilneeeof the National Aweaanunt 
The Secratary ahall raport to the Confrom. the Praaident. and the 
Nation on tha findinfi and raconuaendationa of audi raviewa. The 
r n rnmi aaio ner ahall eonaider the fiiidiiMp and nuMunandationa in 
draioiinr the oo«p«tition to aalact tha oiiaAiiation thrat«h which 
the Office camea out the National Aaeemment 

".B) Thtt Commiaaionar ahall, not later than 6 montha after the 
date of enactment of the National Aaeaaament of Educational 
PragTMa Improvement Act, puUiah a rapoit aattinf forth plane for 
tha odlection of data for tha IMO aaaaaament and pbuia for includ- 
ing other aubjoct araaa in the 1992 and later aaaeaementa. The report 
ehall include methoda by which the reeulta of the National Ameaa- 
ment of Educational Program may be raported ao that the reeuiU 

EMC 5S 
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^ mora roMlily availabla and more eaeily underatood by educators, 

fiflS'SSIIL^J^LST^ f^f • "»«**^ »»y "hich itemi; 
«ill be /Wfiewad to identify and eieiuda itema which reflect racial, 
ejWural, gmder, or .regional biaa. Tha report ahall be developed 

fSTrfS^'SS ."St •^»*«» officiala.^. 
bera of tha Board appomted under pari«raph (5), and the general 

J?S2!l!i?*i'^J'^ "^.Hlf Public 
*il2SS^^l^iIS^J'!!!^ P^**^ *"»• ■l*n»Priato Infant 

SMOMitaiiMdirSM wcommendn. 

^l^-'T l'P'yyy*' or. AfyorMATiOHa.-8action 406 of the Act ia aou8Ci22i. 
ommM ^ ateikug out auhaaction (e) and by redeak^ating aubaao- 
lioaa(Oand<i)MfaeactionB(e)and(a raapectivefy!^^ 

(c) Ranvanoif or PtiNoa pm AMmaMiara-4l) Ssctinn 

•mcndodto raad aa roUowa: 

KD) Not lam than |»^,000 for tha fiacal year 1989, and auch 
fkTJT JSS! "/r— 'y for each of the fiecal yeare 1990 
tbrouch 1991. ahaU be avmhdiJe to carry out aection 406(i) of 
thie Act (ralatmg to the National Amimment of Eduction 
iTognm); . 

(2) Section mmm of the Act (m redeaignatod by eubaection 
r.*Jf'l.!".*^'^n**','}y inaerUna a comma and '>icept for subaection 
(I) of that laction." unmadiately after "Act". 
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